Sensitivity – Local index to control chaoticity or gradient globally –

Here, we introduce a fully local index named "sensitivity" for each neuron to control chaoticity or gradient globally in neural network (NN). We also propose learning method adjust it "sensitivity adjustment (SAL)". The is the magnitude of its output with respect inputs. By adjusting time average 1.0 neuron, information transmission changes be moderate without shrinking expanding both forward and backward computations. That results through layer neurons when weights inputs are random. Therefore, SAL can dynamics recurrent NN (RNN). It solve vanishing problem error backpropagation (BP) deep feedforward an RNN. demonstrate that applying RNN small random initial weights, log-sensitivity, which logarithm RMS (root mean square) sensitivity over all neurons, equivalent maximum Lyapunov exponent until reaches 0.0. show works BP BPTT (BP time) avoid 300-layer learns lag 300 steps between first input output. Compared manually fine-tuning spectral radius weight matrix before learning, SAL's continuous nonlinear nature prevents loss sensitivities during resulting significant improvement performance.